Allegheny County Predictive Risk Modeling: Community Meeting 
Q&A Session Notes (April 2016) 
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• What was the performance statistic for mammogram as mentioned earlier? 

• What is the role of implicit bias, values, attitudes (e.g., towards race, class) 
from the call screeners using this tool in their decision-making? 

• What will the call screeners see as a factor in the tool? 

• Will ZIP code be included as a factor? Will the tool be able to identify 
particular ZIP code/areas in need of services, e.g., where there are not 
currently any Family Support Centers? 

• How did ethics play a role in designing the model? 

• How do you negotiate or resolve concerns about potential bias with 
historical administrative data? 

• What are the factors that model will use? 

• Will preventive services be offered on a voluntary basis? 






• Are the data from public systems? 

• How long does the screening score follow the child? 

• How does the predictive risk modeling in DHS align with efforts in simulation 
modeling for decision-making, e.g. in economics where there are clearly 
different impacts ($ vs. human beings)? 

• Where is DHS and predictive risk modeling going in the near future, i.e. 5 
years from now? Is this the first step in a longer progression with predictive 
analytics? 

• What is the difference between 7 years to another 7, did anything change or 
happen historically to skew data? 

• How will the predictive risk modeling be affected by policy updates, 
specifically new laws with investigating time frame and age in child welfare? 

• How do you think parents or guardians will react or understand to this 
predictive risk modeling? Do you think they'll find it acceptable? 

• Can you, or will you, compare call screen staff using the tool vs. not using the 
tool? 

• How much weight is there on ZIP Code? 

• How much will the model evolve during the course of the evaluation? 

• Community Meeting 4/18/2016 

• Audience Questions, Notes for Erin 

• Are schools included (e.g., potential data points include absentee, # schools 
attended)? 

• Does the model include parent substance use? Does it go into detail about 
specific type of substance? 

• Will placement agencies (e.g., placement staff) know the predictive risk score 
when a child enters their placement agency? Will it, or how will it get 
communicated to them? 

• Will you tell families the score? 

• How did you decide on "20"? 

• How will strengths-based nature of child welfare intervention be used when 
in implementing prevention-based efforts using the predictive risk modeling, 
i.e., how will you still be strengths-based once identifying a family or child is 
at high-risk, and communicating the need for preventive services? 
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Is one year enough for the evaluation? (Mary Carrasco, Child’s Place) 

Jeremy responded that no, it’s not entirely enough time to evaluate the tool. But at the 
same time, many changes are constantly being made throughout the system as years 
pass, so even if you waited for additional time for evaluation, you might not be able to 
attribute all changes to the tool. 


Which/how many families were included in the base analyses? Any 
descriptions of them? Are they applicable to Allegheny County? (Doug, Wesley 
Spectrum) 

Rhema responded that the team initially took every OCYF referral from 2007-14 and 




allowed for an exposure window of 2 years. For each referral, all individuals were 
included if they were on the client list for the referral, as well as all of their histories, 
etc. The referrals used for building the tool were only from Allegheny County - the 
modeling is built specifically off of our local historic data and isn’t valid for (or derived 
from) any other localities. Emily added that the tool won’t be static in the long run: 
since it’s built on our own data, it can be adjusted and change as the county changes. 


How will situational context/clinical judgment fit into the model? (e.g., what if 
the mother has moved away, etc.) (Laura Mulvey, OCD) 

The model will be influenced by who is included on the referral and their histories. In 
that particular example, the mother-in-law’s departure should impact the score, as 
would a new boyfriend joining the case, (or any other situation where the client list is 
changed). 


Can the screening tool always influence screening decisions? 

No; the tool will only apply to GPS cases. CPS cases are always screened in and won’t be 
impacted by the tool’s findings. 


Will we communicate with the state about cases passed down to us? 

It is possible that we could use the tool’s findings in the future to educate the state on 
what we find. 


Have other places seen any unintended negative consequences? (Ellie 
Newman, County Mgr’s office) 

There aren’t any real comparisons yet. But we will want to monitor this closely and 
understand the tool’s impacts as they happen through our process and impact 
evaluations. 


Will the tool cause a bias against children that have more data? (Doug Henry, 
Western Psych) 

Rhema responded that there will be a minimum threshold of data completeness for the 
score to generate. Children with little or no data at all won’t receive a score, so the 
scores will be built to assess the risks among families/children with enough data to 
assess. 


Is a higher volume of referrals expected? (Teresa,_) 

This is some we will want to consider. For any additional cases being screened in based 
on higher scores, ideally there will also be some effect of referrals with lower scores 
being screened in at a lower rate to offset it. From the provider standpoint, the effects 
in referral load are downstream. 



What if administrative data doesn't capture the family accurately? What 
about complex situations? 

[I didn’t really capture our response here] 


What about racial disproportionality? 

• On the impact side, we’re interested in the overall accuracy level of the tool, as 
well as the accuracy between and within subgroups. The evaluation will be able 
to assess whether the tool is accurate ("good’’ or "bad"), as well as specific 
focuses for improvement. 

• Ultimately the tool may not be able to correct any existing biases, but its use 
should make them and the decision processes more measurable/transparent. 



